Controlling graphic field of view by operating head mounted display

ABSTRACT

An immersive head-mounted display (IHMD) can include motion sensing detectors and a display. The IHMD can generate a first generate a first graphical field of view to present on the display. The IHMD can generate a request for a user input. The IHMD can generate one or more second graphical fields of view that are shifted in various directions from the first graphical field of view to present on the displays in response to receiving movement data, from the motion sensing detectors, indicative of the IHMD being moved in the various directions. The IHMD can analyze the movement data to determine a user reply to the request in response to receiving the movement data within a predetermined amount of time. The IHMD can determine the movement data corresponds to an affirmative head shake and perform an affirmative action or corresponds to a negative head shake and perform a negative action.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/404,165, entitled “CONTROLLING GRAPHIC FIELD OF VIEW BY OPERATINGHEAD MOUNTED DISPLAY,” and filed on Jan. 11, 2017, and which is herebyincorporated by reference in its entirety.

FIELD

The present application is directed to immersive head-mounted displaysthat are operable to present graphical information to a user.

BACKGROUND

A head-mounted display (HMD) is a display device that is worn on thehead of a user. The HMD can be operable to have either a unitary displayor two separate displays, that is one display for each eye. The HMDdisplay can be arranged to allow light into the field of vision of theview. In at least one example, the HMD can be an immersive HMD (IHMD).An IHMD is operable to prevent light outside of the IHMD from enteringinto the field of view. The amount of light that is prevented fromentering into the field of view can be dependent upon the fit of theIHMD against the user's face. In at least one example, the IHMD includesa gasket like interface that is designed to contact the user's face.

The IHMD can be configured to display primary content to a user. Theprimary content can include one or more of virtual reality data,augmented reality data, computer animation data, video data, movie data,or game data.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations of the present technology will now be described, by wayof example only, with reference to the attached figures.

FIG. 1 illustrates an example of a system in which an IHMD is operable.

FIG. 2 illustrates an example of an IHMD system.

FIG. 3 illustrates an example of a block diagram of a control device.

FIG. 4 illustrates an example of a block diagram of an IHMD.

FIG. 5 illustrates a graphical user interface shown on a display of theIHMD.

FIG. 6A illustrates a graphical user interface shown on a display of theIHMD having an electronic appliance content display.

FIG. 6B illustrates a graphical user interface shown on display of theIHMD having an alert notification.

FIG. 6C illustrates setting data of an electronic appliance contentdisplay.

FIG. 7A illustrates a graphical user interface of an IHMD.

FIG. 7B illustrates a graphical user interface of an IHMD having adownwardly modified graphical field of view.

FIG. 7C illustrates a graphical user interface of an IHMD having agraphical field of view.

FIG. 7D illustrates a graphical user interface of an IHMD having aleftward modified graphical field of view.

FIG. 7E illustrates a graphical user interface of an IHMD havinggraphical field of view.

FIG. 7F illustrates a graphical user interface of an IHMD having apartially leftward modified graphical view of view.

FIG. 7G illustrates a graphical user interface of an IHMD having amodified graphical field of view.

FIG. 8 illustrates a flowchart of an example method implementing anIHMD.

DETAILED DESCRIPTION

It will be appreciated that for simplicity and clarity of illustration,where appropriate, reference numerals have been repeated among thedifferent figures to indicate corresponding or analogous elements. Inaddition, numerous specific details are set forth in order to provide athorough understanding of the embodiments described herein. However, itwill be understood by those of ordinary skill in the art that theembodiments described herein can be practiced without these specificdetails. In other instances, methods, procedures and components have notbeen described in detail so as not to obscure the related relevantfeature being described. Also, the description is not to be consideredas limiting the scope of the embodiments described herein. The drawingsare not necessarily to scale and the proportions of certain parts havebeen exaggerated to better illustrate details and features of thepresent disclosure.

Several definitions that apply throughout this disclosure will now bepresented. The term “coupled” is defined as connected, whether directlyor indirectly through intervening components, and is not necessarilylimited to physical connections. The connection can be such that theobjects are permanently connected or releasably connected. The term“substantially” is defined to be essentially conforming to theparticular dimension, shape or other word that substantially modifies,such that the component need not be exact. For example, substantiallycylindrical means that the object resembles a cylinder, but can have oneor more deviations from a true cylinder. The term “electronic appliance”is defined as any electronic device, household or commercial, that canbe communicatively coupled to one or more systems. For example, anelectronic appliance can be an air conditioning system, heating system,oven, a stove top, a refrigerator, a garage door opener, a gate opener,a light switch or lighting system, a smart phone, a tablet computer, alock box, a gun safe, a doorway sensor, a motion sensor, or any othercommunicatively couplable electronic device.

The term “refrigerator” can refer to an electronic appliance capable ofstore and cooling food. A refrigerator can refer a common appliancehaving both a cold storage portion, and a frozen storage portion. Theterm refrigerator can also refer to an appliance having only a coldstorage portion or only a frozen storage portion. The term “smartphone”can refer to a handheld electronic device configured to send and/orreceive data. The data can be sent over a Wi-Fi or cellular network. Thesmartphone can send and/or receive data relating to telephone calls,short message service (SMS) messages, electronic messages, electronicmail (e-mail), one or more applications, including games and webbrowsers. The term “air conditioner” can refer to an electronicappliance capable of cooling a building, home, or other space. The airconditioner can include a cooling coil, an air movement fan, and athermostat. In at least one example, the air conditioner can beconfigured as a heat pump to heat a space. The term “graphical userinterface” means visual data provided to the user. The graphical userinterface can provide a game, a movie, a menu, or other visual data.

The present disclosure describes an immersive head-mounted display(IHMD) and IHMD system. The IHMD system is operable to allow the IHMD tocommunicate with one or more external devices. In other examples, theIHMD itself can be operable to communicate directly with one or moreexternal devices. The IHMD is operable to receive data from one or moreremote systems and further operable to receive data and/or input from atleast a controller in response to the one or more remote systems. Inother examples, the controller can be integrated into the IHMD.

The IHMD system can include a control device communicatively coupled toone or more external devices. The external devices can be electronicappliances, or more specifically household electronic devices such as arefrigerator, a smart phone, an air conditioner unit, a heater, a stove,a dishwasher, or the like. The IHMD can similarly be communicativelycoupled to the control device. The control device can send and/orreceive data between the IHMD, the IHMD system, and the one or moreexternal devices. The IHMD can include a hardware data receivingcomponent operable to receive data generated by the one or more externaldevices from the control device. The IHMD can also include a processingunit having one or more processors communicatively coupled to thehardware data receiving component, and a display device coupled to theprocessing unit so as to generate a graphical user interface featuringprimary content and external device content. The external device contentcan identify the one or more external devices and provides one or morecontrol elements associated therewith. The external device content caninclude settings, alerts, or other notifications related to the one moreor more external devices.

The processing unit can modify the graphical user interface to displaymodifiable settings of a selected external device and update themodified graphical user interface based on requested changes to themodifiable settings. The processing unit can also transmit setting dataobtained from the modified settings to the selected external device froma hardware transmitting component to the control device.

In some instances, the control device can communicatively couple with aninput device configured to select the one or more external devices,settings, or alerts. The input device can be a user manipulated device,such as a motion sensing controller detecting user movements as inputs.The input device can also be one or more retinal cameras monitoring auser's eye movements.

The IHMD system can provide a user an IHMD having a display devicepresenting a graphical user interface featuring primary content andexternal device content. The display device can be communicativelycoupled to a control device and the control device can also becommunicatively coupled with one or more external devices. The user canselect and modify settings of the external devices displayed within thedisplay device. A processing unit having one or more processorscommunicatively coupled with the control device can modify the graphicaluser interface and update the modified graphical user interface based onthe request changes and transmit the setting data to the external devicevia the control device.

The IHMD system can communicatively couple, directly or through anintervening control device, with one or more electronic appliances suchthat an IHMD display can provide content received from the one or moreelectronic appliances and allow the IHMD system to modify or respond tothe one or more electronic appliances in view of the received content.In some instances, the received content can be an incoming phone callfrom a smartphone and providing options to answer the phone call orignore the phone call. In other instances, the received content can bean alert a refrigerator door is open. The IHMD system can couple withone or more controllers providing input to the IHMD system to generate amodification or response to the one or more electronic appliances.

An IHMD system method can utilize the elements as described above andimplemented collectively to form the system. The IHMD system cangenerate a graphical user interface having primary content. Thegraphical user interface can be generated by a control device, eitherseparate from the IHMD 100 or integrated therewith, and transmitted tothe IHMD. The graphical user interface can include the primary contentand electronic appliance content received from the one or moreelectronic devices coupled to the system.

The IHMD system can display, on an IHMD, the generated graphical userinterface. The IHMD system can overlay the electronic appliance contentwithin the graphical user interface displayed within the IHMD. Theelectronic appliance content can include information relating to one ormore electronic appliances. In some instances, the electronic appliancecontent can be displayed in response to a user input. In otherinstances, the electronic appliance content can be displayed in responseto receiving data from the one or more electronic appliances.

The IHMD system can receive a selection of one or more electronicappliances and a selection of one or more control elements associatedwith the selected electronic appliance. The control element can be asetting associated with the selected electronic appliance. The selectionof one or more electronic appliances and one or more control elementscan be received by a hardware data receiving component within the IHMDand the selection can be generated by a controller coupled with the IHMDor IHIVID system. For example, the selected electronic appliance can bean air conditioning unit and the selected control element associatedwith the air conditioning unit can be a temperature setting.

The IHMD system can receive a modification of at least one of the one ormore control elements. The one or more control elements can be modifiedwithin the electronic appliance content of the graphical user interface.The modification can be received by a hardware receiving componentwithin the IHMD. For example, the selected control element of atemperature setting can be adjusted to lower the temperature from afirst temperature (for example 22° C.) to a second temperature (forexample 20° C.).

The IHMD system can send a communication, via the hardware datatransmitting component, having setting data to the selected electronicappliance. The setting data can correspond to the modification of thecontrol element received by the IHMD system. For example, thecommunication can have setting data corresponding to lowering thetemperature setting from the first temperature to the secondtemperature.

FIG. 1 illustrates an example of a system 2 in which an IHIVID 100operates. The IHMD 100 can be part of an IHIVID system 10 that includesa control device 200. The control device 200 can be coupled with one ormore electronic appliances 12. The one or more electronic appliances 12can include a refrigerator 30, a smartphone 40, an air conditioner unit20, a heater, a stove, a dishwasher, a microwave oven, a hot waterheater, a washing machine, a dryer, a stove, a door entry alarm, awindow alarm, and/or an oven. The one or more electronic appliances 12can communicate one or more settings, alerts, functions, or otherelements to the IHIVID system 10.

In at least one instance, the refrigerator 30 can communicate thepresent operating settings such as fridge temperature, freezertemperature, power status. The refrigerator 30 can also communicate anoperating status, for example door status, interior light status, mostrecent change in door or light status. In other instances, therefrigerator 30 can communicate relating to the contents of the fridgeportion, the freezer portion, or a combination thereof. In otherinstances, the air conditioner unit 20 can communicate the currentsettings such as temperature, fan setting, heat, cool, and/or thepresent temperature setting and schedule.

The system 2 can include a controller 90 communicatively coupled withthe control device 200 and the IHMD 100, thus providing input to theIHMD 100 and the control device 200. The controller 90 can generateinputs in response to alerts or communications from the one or moreelectronic appliances 12. The controller 90, or input device, can be auser manipulated device, such as a motion sensing controller detectinguser movements as inputs. In another example, the controller 90 can beone or more retinal cameras monitoring a user's eye movements within theIHMD 100. The controller 90 can generate inputs to modify an operatingparameter, status, or alert notification of the one or more electronicappliances 12. In some instances, the controller 90 can manipulate acursor within the IHMD 100. The controller 90 can be coupled to thecontrol device 200 or directly coupled with the IHMD 100.

The control device 200 can communicatively couple with a router 50 andtermination equipment 60. The router 50 and termination equipment 60 canprovide communication between the IHMD 100 and the one or moreelectronic appliances 12 by directing network traffic of the system 2.The router 50 and termination equipment 60 can be communicativelycoupled to the control device 200 via a wired or wireless connection.While the IHMD 100 is illustrated as being coupled with the controldevice 200, the IHMD 100 can be self-contained, such that theperformance and functionality of the control device 200 is integrallyformed within the IHMD 100. Thus, the IHMD 100 can communicate directlywith one or more of the devices illustrated in FIG. 1. For example, theIHMD 100 can be operable to communicate directly with the one or moreelectronic appliances 12. Furthermore, the IHMD 100 can be operable tocommunicate directly with the router 50 and the terminal equipment 60.In the illustrated example, a router 50 and terminal equipment 60 arepresent, however, the terminal equipment 60 can alternatively becombined into a single device that allows communication outside of alocal network.

In at least one example, the control device 200 can be configured tocommunicate with a plurality of external devices, which includes theelectronic appliances 12, air conditioner 20, refrigerator 30,smartphone 40, router 50, terminal equipment 60, lighting equipment 70,intercom equipment 80 and one or more of the above described componentsof the system. When the control device 200 is coupled to the terminalequipment 60, the control device 200 can receive data from one or moreremote data storage devices, which can be at one or more locations onthe internet. The remote data storage can include cloud storage, datacenters, off-site storage, or a server within the local area network(LAN).

The system 2 can communicatively couple the IHMD system 10 with the oneor more electronic appliances 12, such as a refrigerator 30. In one suchinstance, the refrigerator 30 can include one or more of an RFID scanner32, a camera 34, a light 36, display screen 38, or a bar code scanner39. The refrigerator 30 can utilize the camera 34 to provide the IHMDsystem 10 a real-time view of the contents within the refrigerator 30.In at least one example, the refrigerator 30 can implement productrecognition technology to formulate a list of items present within therefrigerator 30. The light 36 can provide the system 2 with informationrelating to the on/off status of the light, such that the IHMD 100 candisplay the status. The light 36 can further provide informationrelating to the functionality and alert if the bulb requiresmaintenance. The display screen 38 can provide information at therefrigerator relating to contents, operating temperatures, and otherfunctions. The display screen 38 can further allow input of informationsuch as contents or operating settings that can be communicated to theIHMD 100. The barcode scanner 39 can allow scanning of one or moreproducts placed in the refrigerator 30. The scanned product can bedisplayed on the display screen 38 or communicated to the IHMD 100.

FIG. 2 illustrates an IHMD system 10 having an IHMD 100 communicativelycoupled to one or more electronic appliances. The IHMD 100 cancommunicatively couple directly with a plurality of electronicappliances 12. As illustrated the IHMD 100 is directly communicativelycoupled to an air conditioner 20, a smartphone 40, a controller 90, agarage door opener 95, and refrigerator 30. In some instances, thecontrol device 200 can be integrated within the IHMD 100, thuseliminating the need for a separate control device 200.

The IHMD 100 can include one or more tracking elements 130 disposedwithin or on a headset 110. The tracking elements 130 can be detected bya camera to determine the position and orientation of the headset 110relative to the camera. The tracking elements 130 can be phosphorescentelements detectable by the camera. The camera can be implemented ascamera 120 shown in FIG. 1, or as any other camera coupled to the system2.

In some instances, each of the one or more tracking elements 130 canprovide a different intensity, size, shape, or wavelength of light, thusallowing the camera 120 to determine the position, orientation, andmovement of the headset 110. In other instances, the tracking element130 can be formed in distinctive patterns, thus allowing the camera 120to determine position, orientation, and movement of the headset 110. Inat least one example, the tracking elements 130 can be used to detectthe movement of the headset in combination with one or more gyroscopesor accelerometers located within the IHMD 100. In another example, thetracking elements 130 can be used to calibrate the one or moregyroscopes or accelerometers located within the IHMD 100.

FIG. 3 illustrates an example of a block diagram of a control device.The control device 200 can include a wireless adapter 202, an IHMDconnection 204, and a controller adapter 206. The wireless adapter 202communicatively couples the control device 200 with the router 50,termination equipment 60, and/or the electronic appliances 12. Thewireless adapter 202 can communicatively couple the control device 200via Wi-Fi, Bluetooth™, Z-Wave, cellular, or any other wireless datatransfer protocol.

The IHMD connection 204 communicatively couples the control device 200with the IHMD 100. The IHMD connection 204 allows data transfer betweenthe control device 200 and the IHMD 100. The IHMD connection 204 cancommunicatively couple the IHMD 100 via a wired connection or a wirelessconnection, such as Wi-Fi, Bluetooth™ cellular or any other wirelessdata transfer protocol.

The controller adapter 206 communicatively couples the control device200 with a controller 90 capable of receiving one or more inputs. Asdescribed above, the controller 90 can be a joystick, keypad, motioncontroller having one or more gyroscopes, or any other user inputdevice.

The control device 200 can further include memory 208, such as RandomAccess Memory (RAM) or other temporary access memory, and one or morestorage devices 210, such as a hard disk drive (HDD) for semi-permanentor permanent storage. The one or more storage devices 210 can be a HDD,a solid state drive (SSD), a combination thereof, or any other storagedevices. A storage adapter 220 can be used to access the memory 208 andthe one or more storage device 210.

The control device 200 can have a processing unit 230 including one ormore processors 232 and memory 234. The processing unit 230 can becommunicatively coupled to each of the wireless adapter 202, IHMDconnector 204, controller adapter 206, memory 208, one or more storagedevices 210, and storage adapter 220. The memory 234 can be cachememory, such as RAM integrated directly with the one or more processors232.

FIG. 4 illustrates an example of a block diagram of an IHMD. An IHMD 300can include a hardware data receiving component 302 having a hardwaredata receiving component 304 and a hardware data transmitting component306. The hardware data receiving component 304 can be operably coupledto receive data indicative of data generated by one or more electronicdevices 12 (see FIG. 1). The hardware data transmitting component 306can be coupled with a control device 200 to transmit setting data to theselected electronic appliance. The setting data can be data obtainedfrom modified settings input by a user.

The IHMD 300 detailed in FIG. 4 can be substantially similar to the IHMD100 as shown and described in FIG. 4. The IHMD 300 can have a poweradapter 308 to provide electrical power to the IHMD 300 and relatedcomponents. The power adapter 308 can be a re-chargeable battery, suchas one or more lithium-ion cells, one or more nickel-cadmium cells, orcan be a wired power source. The power adapter 308 can be coupleddirectly to an electrical source or can be coupled with and receivepower from the control device 200. When the power adapter 308 is are-chargeable battery, the power adapter 308 can include a charging portas well.

A hardware detection device 310 can have one or more gyroscopes 312 andone or more ports 314. The one or more gyroscope 312 can provide theIHMD 300 data relating to positioning, orientation, and/or movement ofthe IHMD 300. The gyroscope 312 can be a gyroscope having a single,three-axis gyroscope chip. The gyroscope 312 can be a plurality ofgyroscope chips. In some instances, the IHMD 300 can have one gyroscopedetecting and providing data relating to movement along a single axis,such as the X, horizontal, axis. In other instances, the IHMD 300 canhave a plurality of gyroscopes detecting movement along multiple axes,such as the X, Y, and Z axes, thus allowing the hardware detectiondevice 310 to track three dimensional movement of the IHMD 300. Each ofthe plurality of gyroscopes 312 can track a single axis, butcollectively, the plurality of gyroscopes 312 can track movement,position, and orientation of the IHMD 300 along multiple axes. Forexample, a first gyroscope 312 can track movement and orientationrelative to the X-axis, a second gyroscope 312 can track movement andorientation relative to the Y-axis, and a third gyroscope 312 can trackmovement and orientation relative to the Z-axis. While gyroscopes havebeen described above and illustrated, the IHMD 300 can implementaccelerometers or other motion sensing detectors. The motion sensingdetectors can be configured to detect motion in a given direction. Inthose situations where the motion sensing detectors are only able todetect motion in a single direction, the motion sensing detectors can bearranged orthogonal to one another to allow sensing in at least threedifferent directions. In some examples, there can be nine or more motionsensing detectors.

The one or more port 314 can couple the hardware detecting device 310with one or more external components to provide additional data andinformation relating to the positioning and movement of the IHMD 300. Inat least one example, the hardware detecting device 310 can include bothone or more gyroscopes and one or more external components. In at leastone example, the one or more external components can include a camera.Additionally, IHMD can be configured to have tracking elements thereonor therein. The tracking elements can be monitored by the camera orother device configured to determine the orientation of the trackingelements. In at least one example, the one or more external componentscan be configured to calibrate the one or more gyroscope. As mentionedabove, the present disclosure contemplates the use of other motionsensing detectors in place of the one or more gyroscope.

A processing unit 316 can have one or more processors 318communicatively coupled to the hardware data receiving component 302.The processing unit 316 can receive data via the hardware data receivingcomponent 304 and can transmit data via the hardware data transmittingcomponent 306. The processing unit 316 can be communicatively coupledwith the power adapter 308 and the hardware detection device 310. Thehardware detection device 310 can send data relating to movement of theIHMD 300 to the processing unit 316. The processing unit 316 can includememory 320 dedicated to the one or more processors 318, such as cachememory.

A display device 322 can be coupled to the processing unit 316. Thedisplay device 322 can be operable to generate a graphical userinterface featuring primary content and electronic appliance content, anexample is as detailed below with respect to FIG. 5. The electronicappliance content can identify the one or more electronic appliances andprovide one or more control elements associated therewith. The displaydevice 322 can be an liquid crystal display (LCD), an light emittingdiode (LED) display, or any other known display technology. In someinstances, the display device 322 can include three dimensional displaytechnology and provide a first eye with a first display, and a secondeye with a second display forming the perception of depth within thecombined display.

The processing unit 316 can operably modify the graphical user interfacebased on the selection of one or more electronic appliances. Theselection of one or more electronic appliances can be received by thehardware detection device 310 in response to the graphical userinterface displayed by the display device 322. The modified graphicaluser interface can display modifiable settings of the selected one ormore electronic appliances. The processing unit 316 can update themodified graphical user interface based on the requested changes to themodifiable settings. Setting data can then be transmitted via thehardware transmitting component 314 to the control device 200. Thesetting data can be obtained from the modified settings.

In at least one instance, the IHMD 300 receives data generated by one ormore electronic appliance and the display device 322 can generate agraphical user interface displaying the data generated by the one ormore electronic appliance. The IHMD 300 and display device 322 canprovide one or more control elements associated therewith. The IHMD 300can receive a selection of one or more electronic appliances and thedisplay device 322 can generate a graphical user interface displayingmodifiable settings of the selected one or more electronic appliances.The IHMD 300 can then receive a requested change to the modifiablesettings and the IHMD 300 can communicate the requested change to theone or more electronic appliance.

FIG. 5 illustrates a graphical user interface shown on a display of theIHMD. The graphical user interface 400 can be generated by an IHMD, suchas the one described above with respect to FIGS. 1 and 4. The graphicaluser interface 400 can have primary content 402 and electronic appliancecontent 404. The primary content can be one or more of a virtualreality, augmented reality, computer animation, video, movie, game,and/or any combination thereof.

The primary content 402 can be paused, stopped, or otherwise interruptedwhen the electronic appliance content 404 is displayed within the IHMD300. The electronic appliance content 404 can appear as an alert to theIHMD 300, thus pausing the primary content 402 until an action is takenrelating to the alert. In other instances, the primary content 402 canplay in the background when the electronic appliance content 404 isdisplayed. An alert priority can be set to allowing pausing orinterruption of the primary content 402 if the alert exceeds a certainpredetermined threshold. For example, the IHMD 300 can pause orinterrupt the primary content 402 and display the electronic appliancecontent 404 if a smartphone receives a call from a phone number within acontact list, but the primary content can remain uninterrupted if thecall received is from a phone number not within the contact list.

In some instances, the electronic appliance content 404 can be watermarkoverlaid on the primary content 402, such that the primary content 402is unobstructed by the electronic appliance content 404. In otherinstances, the electronic appliance content 404 can be overlaid on theprimary content 402, such that the electronic appliance content 404obstructs the primary content 402.

The electronic appliance content 404 can identify one or more electronicappliances 406 and provide one or more control elements 408 associatedtherewith. The one or more electronic appliances 406 can be one or moreof a refrigerator, a smartphone, an air conditioner, a heater, a stove,a dishwasher, a microwave oven, a hot water heater, a washing machine, adryer, a stove, a door entry alarm, a window alarm, an oven, a lightcontrol, or any other electronic appliance.

The electronic appliance content 404 can be changed or modified using avirtual reality controller 450 operated by a controller as describedabove with respect to FIG. 1. The controller can allow manipulation ofthe one or more control elements 408 to view, modify, or change thecontrol elements 408 within the electronic appliance content. Thevirtual reality controller 450 can allow interaction and manipulation ofthe electronic appliance content 404 within the IHMD 300.

The electronic appliance content 404 can be displayed after receiving acommand from the IHMD 300 or control device 200. In some instances, theelectronic appliance content 404 can be dynamically displayed uponreceiving an alert from the one or more electronic appliances 406. Theelectronic appliance content 404 can be overlaid on the primary content402 upon receiving an alert. In other instances, the electronicappliance content 404 can be statically displayed upon a commandreceived by the IHMD 300 and/or control device 200.

The electronic appliance content 404 can display information related toan air conditioner 410 and at least one of power status 412, atemperature setting 414, a timer setting 416, current temperature 418,and current humidity 420. The information related to the air conditioner410 can provide interactive options and non-interactive options. In someinstances, the power status 412 can provide an interactive toggle switchbetween an on/off allowing the operating status 412 to be adjusted. Theinteractive toggle can include a higher temperature graphic in a form ofan upward pointing arrow and a lower temperature graphic in the form ofa downward pointing arrow. In other instances, the operating status canbe a non-interactive such that the operating status 412 is displayed,but cannot be adjusted. Similarly, each of the temperature setting 414,timer setting 416, current temperature 418, and current humidity 420 canbe displayed as interactive options and/or non-interactive options.

The selection of a higher temperature graphic can cause the displayedtemperature setting 414 to be increased and cause the transmission ofsetting data to the air conditioner to adjust the temperature of the airconditioner, accordingly. Selection of a lower temperature graphic cancause the displayed temperature to be decreased and cause thetransmission of setting data to the air conditioner to adjust thetemperature of the air conditioner, accordingly. Additionally, thepresent disclosure contemplates interacting between the menus shown suchas the temperature setting 414 in a manner similar to the discussionbelow with respect to FIGS. 7A-7G.

In other instances, the temperature setting 414 can be selectable andupon selection of the temperature microphone data, obtained from amicrophone communicatively coupled with the IHMD 300 or the controldevice 200. The microphone data can be a spoken new temperature andsetting data is transmitted to the air conditioner to adjust thetemperature accordingly.

The electronic appliance content 404 can display information related tolighting equipment 422 and a toggle switch 424 adjusting the operatingsettings of the lighting equipment. The toggle switch 424 can adjust thelighting equipment between an on operating setting and an off operatingsetting within the graphical user interface 400.

The electronic appliance content 404 can display information relating toa refrigerator 426 including freezer compartment 428, refrigeratorcompartment 430, and/or vegetable compartment 432. In some instances,the refrigerator 426 can also provide information relating to powerstatus, temperature setting, current temperature, last time therefrigerator was open, interior light status, such as on or off, anddoor status, such as open or close.

The refrigerator 426 can provide information relating to the freezercompartment 428, for example contents of frozen vegetables (“A”), ice(“B”), and ice cream (“C”). Similarly, the refrigerator 426 can provideinformation relating to the contents of refrigerator compartment 430,such as milk and eggs, and information relating to the contents of thevegetable compartment 432. For example, the IHMD 300 can includeinformation from the refrigerator 426 indicating the vegetablecompartment 432 presently contains cabbage, carrots, and potatoes. Insome instances, the information relating to the freezer compartment 428,refrigerator compartment 430, and vegetable compartment 432 can benon-interactive, static displays. The non-interactive, static di splaysprovide information within the IHMD 300, but cannot be modified orchanged using a controller 90. The dynamic displays provided within theIHMD 300 can contain information that can be modified or changed usingthe IHMD 300 and controller 90.

As described above with respect to the air conditioner 410, therefrigerator can similarly have interactive and non-interactive options.More specifically, the refrigerator can have an interactive selectablegraphic to adjust the temperature of the freezer compartment,refrigerator compartment, vegetable compartment, or any combination ofcompartments within the refrigerator 426. The selectable graphic caninclude a higher temperature graphic in a form of an upward pointingarrow and a lower temperature graphic in the form of a downward pointingarrow. The selection of a higher temperature graphic can cause thetemperature setting 414 to be increased and cause the transmission ofsetting data to the refrigerator to adjust the temperature of therespective refrigerator compartment, accordingly. Selection of a lowertemperature graphic can cause the displayed temperature to be decreasedand cause the transmission of setting data to the air conditioner toadjust the temperature of the respective refrigerator compartmentaccordingly.

The electronic appliance content 404 can further include informationrelating to an intercom 434. The information relating to the intercom434 can have one or more toggle switch 436. Information relating to asmartphone 438 can similarly have one or more toggle switches 440, 442relating to phone calls and emails, respectively.

FIG. 5 provides an example of the interactivity and alert datainteractivity, the present disclosure includes different arrangementsthat are beyond what is described in FIG. 5 based on the totality of theremainder of the disclosure. The interactivity and alert datainteractivity can be displayed and arranged beyond what is shown anddescribed above in FIG. 5. The electronic appliance content 404 can bedisplayed on the left edge, right edge, bottom edge, top edge, oroverlaid on the entire display. Additionally, the electronic appliancecontent 404 can provide content to a plurality of electronic appliancesat a time (as shown), or provide content to only a single electronicappliance at a given time and allow a user to scroll through individualelectronic appliances.

FIG. 6A illustrates a graphical user interface of an IHMD. The graphicaluser interface 400 can have primary content 402 and electronic appliancecontent 404. The electronic appliance content 404 can be overlaid over abottom portion of the primary content 402. The electronic appliancecontent 404 can provide alert information 460 relating to one or moresettings or control elements of the selected electronic appliance. Forexample, a refrigerator status 462 can overlay at least a portion of theprimary content 402 and provide information relating to one or more of apower status 464, a light status 466, a door opened status 468, a doorstatus 470, contents status 472, and a temperature status 474.

The power status 464 can indicate whether the refrigerator 462 is on oroff, and in some instances provide the ability to alter the powerstatus. The light status 466 can indicate whether the lighting systemwithin the refrigerator 462 is on or off, and in some instances canindicate if one or more of the lighting elements needs to be replaced.The door opened status 468 can indicate the amount of time between thepresent time and the most recent door open event. The door open status468 can indicate the refrigerator 462 was most recently opened, forexample 2 hours ago, or any other amount of time that the door wasopened. The door status 470 can indicate the present state of therefrigerator door. In some instances, the door status 470 can indicatethe refrigerator 462 was not properly closed and the door remains open.The contents status 472 can provide information relating to the currentcontents of the refrigerator 472. In some instances, the contents can bedetermined by one or more cameras within the refrigerator 462 asdescribed above with respect to FIG. 1. The temperature status 474 canprovide information relating to the present temperature of therefrigerator 462, and in some instances allow modification of thetemperature setting to lower the temperature or increase the temperaturesetting 474.

FIG. 6B illustrates a graphical user interface of an IHMD having analert status. The graphical user interface 400 displayed within the IHMD300 can have primary content 402 and alert information 460. The alertinformation 460 can indicate a refrigerator 462 having a temperaturestatus 474 that exceeds a predetermined setting. The predeterminedsetting can be a specific temperature, or a temperature range, and therefrigerator 462 can generate an alert 462 to be displayed within thegraphical user interface 400 when the temperature differs from thepredetermined setting. For example, the predetermined temperaturesetting can be 4° C. as illustrated in FIG. 6A, but the graphical userinterface 400 can provide an alert 462 indicating the temperature status474 within the refrigerator 462 exceeds the setting.

While the illustrated example in FIG. 6B is drawn to a refrigerator, thealert information 460 can be received by and pertain to any one or moreof the electronic devices 12 coupled with the system 2.

FIG. 6C illustrates example setting data of an IHMD system. Thegraphical user interface 400 can display modifiable settings 490 of therefrigerator 462. The modifiable settings can include a door open alert476 and a temperature alert 478. The door alert 476 can provideinformation relating to how long the refrigerator door has been open.The door alert 476 can have a modifiable setting for a predeterminedtime 480 and a priority 482. The predetermined time 480 can allow asetting of how many minutes the door can be ajar before an alert isgenerated. For example, the predetermined time 480 can be set to “3minutes” and if the door is open for more than 3 minutes, an alert willbe generated. The priority 482 allows a setting to determine thepriority of the alert generated, for example “high,” “medium,” or “low.”The priority 482 can determine whether the alert interrupts the primarycontent 402 of the graphical user interface 400. In at least oneinstance, a priority 482 setting of “high” will interrupt and pause theprimary content 402 to display the alert, a priority setting 482 of“medium” will display the alert content overlaid on the primary content402 without interrupting or pausing, and a priority 482 of “low” willdisplay an alert icon or other notification indicating an alert exists,without otherwise obscuring the primary content 402.

For example, the temperature alert 478 can be set to a predeterminedtemperature 484 and a “high” alert status 486 indicating the primarycontent 402 should be interrupted and paused should the refrigeratordetermine the temperature deviates from the predetermined temperature484.

The setting data for the refrigerator 490 can be modified from the IHMD300 to alter the alert priority or modify the predetermined time 480 ofdoor alert status 476 or the predetermined temperature 484 of thetemperature alert 478. The IHMD 300 communicate the modifications to theselected electronic appliance.

FIG. 7A illustrates a graphical user interface of an IHMD. The graphicaluser interface 500 can have a graphic field of view 502 generated usingat least a portion of graphical data received by the IHMD. The IHMD canbe an IHMD 300 as described above with respect to FIG. 4. The graphicalfield of view can be modified based on data received from one or moregyroscope 312 coupled with the IHMD 300. The modified graphical field ofview can allow the graphical user interface to vary in response tomovement, position, and/or orientation of the IHMD 300.

The processing unit 316 can receive an input from an input interfacedirected toward the specific target 504. In some instances, the inputinterface is operable to be communicatively coupled to the processingunit 316 and configured to detect spoken questions or commands. In otherinstances, the input can be motion of the IHMD 300 in a specificdirection or orientation to be detected by the one or more gyroscopes312.

The processing unit 316 of the IHMD 300 can identify a specific target504 from one or more potential targets within the graphical field ofview 502. The specific target 504 can alternatively be identified withinthe modified graphical field of view. The identification of the specifictarget 504 can include determining interaction objections 506 locatedwithin the graphical field of view 502. The processing unit 316 canselect one or more interaction objects 506 displayed within a centralregion or a majority of the graphical field of view 502. In someinstances, the one or more interaction objects 506 can be a person, arobot, an animal, a computer, or a kiosk.

The processing unit 316 can be operable to generate response data basedon the received input by interface, such as a spoken question orcommand. The response data can be audio data, such as a spoke response,or a graphical textual data, such as a display written response. Theresponse data can be selection from a predetermined group of responsedata or the response data can generate new graphical data used togenerate the graphical user interface 500.

FIG. 7B illustrates a graphical user interface of an 11-IMD with amodified graphical field of view. Movement of the 11-IMD 300 cangenerate a modified graphical field of view 508. The modified graphicalfield of view 508 can generated as an input to the specific target 504.In some instances, the modified graphical field of view 508 can bemovement of the 11-IMD 300 in an up or down direction generated as aninput or response to the specific target 504 or interaction object 506.

FIG. 7C illustrates a graphical user interface of an 11-IMD returning tothe graphical field of view. The graphical field of view 502 can returnto the portion of the graphical user interface 500 after the input orresponse to the specific target 504 is generated.

FIGS. 7A-7C can illustrate an 11-IMD input of a “yes” command in view ofresponse data generated by the processing unit 316. For example, theresponse data can be posed in the form of a question requiring an IHMDinput. Movement of the 11-IMD 300 in a vertical, up and down, motion canbe detected by the one or more gyroscopes 312 as a responsecorresponding to “yes” or affirmative answer. The processing unit cantransmit a downward shifted graphical field of view and an upwardshifted graphical field of view in response to a head shake movement inthe downward and upward direction. In some instances, the affirmative,or “yes” command, can be made by moving the upward and then downward.The movement of the 11-IMD 300 can be further detected by one or more ofthe motion sensing devices as described above including but not limitedto a camera, tracking elements, and accelerometers.

While a vertical, up and down, movement of the graphical field of viewis illustrated as the input or response, it is within the scope of thisdisclosure that the movement of the graphical field of view can be in ahorizontal direction, left to right. FIGS. 7D-7F can collectivelyillustrate an IHMD input of a “no” or negative command in view ofresponse data generated by the processing unit 316. For example, theresponse data can be posed in the form of a question requiring an IHMDinput. Movement of the IHMD 300 in a horizontal, left and right, motioncan be detected by the one or more gyroscopes 312 as a responsecorresponding to “no” or negative answer. The movement data can indicatea headshake in a left and right direction. The processing unit cantransmit a leftward shift in graphical field of view and a rightwardshift in graphical field of view in response to a head shake in theleftward and rightward direction. In some instances, the head shake canbegin by moving rightward and then leftward.

FIG. 7G illustrates a graphical user interface of an IHMD acquiring anew target. The graphical user interface 500 can provide multiplespecific targets 506, including one or more specific targets beyond thegraphical field of view 502. The graphical field of view 502 can bemodified to seek and/or acquire a new specific target 510. The graphicalfield of view 502 can be modified through movement of the IHMD 300 orinput through a controller. The movement data can indicate movement in ameasured direction away from the graphical field of view, such thatvarying graphical field views are displayed until a designated target isdisplayed within one of the varying graphical field of views and whenmotion of the IHMD is stationary got more than a predetermined amount oftime.

As can be appreciated in FIG. 7G, the graphical field of view 502 ismoved right within the graphical user interface 500 and acquires a newspecific target 510. The new specific target 510 can then elicit aninput or response from the processing unit 316.

FIG. 8 illustrates a flowchart of an IHMD system. An IHMD system method700 can utilize the elements as described above and implementedcollectively to form the system 2. The IHMD system method 700 caninclude additional blocks and the arrangement of the blocks as presentedis not intended to be presented as a particular order, as they blockscan be performed in numerous orders. The IHMD system method 700 can bebegin at block 702.

At block 702, the IHMD system generates a graphical field of view withinan IHMD. The graphical field of view can be generated using at least aportion of the graphical data.

At block 704, the graphical field of view can be modified based on datareceived from at least one gyroscope, thus generating a modifiedgraphical field of view.

At block 706, the VIVID system can identify a specific target from aplurality of potential targets within the modified graphical field ofview.

At block 708, the IHMD system receives an input from an input interface.The input can be directed towards the specific target. In someinstances, the input can be a head motion in response to a questionposed by the specific target. In other instances, the input can be anaudio input directed to the specific target.

1. An immersive head-mounted display (IHMD) comprising: one or moremotion sensing detectors; one or more displays; one or more processors;and memory including instructions that, when executed by the one or moreprocessors, cause the one or more processors to: generate a firstgraphical field of view to present on the one or more displays; generatea request for a user input; generate one or more second graphical fieldsof view that are shifted in one or more directions from the firstgraphical field of view to present on the one or more displays inresponse to receiving movement data, from the one or more motion sensingdetectors, indicative of the IHMD being moved in the one or moredirections; and analyze the movement data to determine a user reply tothe request in response to receiving the movement data within apredetermined amount of time.
 2. The IHMD of claim 1, further comprisinginstructions that when executed by the one or more processors furthercause the IHMD to: determine the movement data corresponds to anaffirmative head shake; and perform an affirmative action responsive tothe request.
 3. The IHMD of claim 1, further comprising instructionsthat when executed by the one or more processors further cause the IHMDto: determine the movement data corresponds to a negative head shake;and perform a negative action responsive to the request.
 4. The IHMD ofclaim 1, further comprising instructions that when executed by the oneor more processors further cause the IHMD to: determine the movementdata does not correspond to an affirmative head shake or a negative headshake; and generate a second request for the user input.
 5. The IHMD ofclaim 1, wherein the request comprises data requesting an answer to aquestion from an interaction object.
 6. The IHMD of claim 1, wherein therequest comprises data requesting to receive or ignore alert data froman electronic appliance.
 7. The IHMD of claim 1, wherein the requestcomprises a modifiable setting of an electronic appliance.
 8. Acomputer-implemented method, comprising: generating, by one or moreprocessors of an immersive head-mounted display (IHMD), a firstgraphical field of view to present on one or more displays of the IHMD;generating, by the one or more processors, a request for a user input;generating, by the one or more processors, one or more second graphicalfields of view that are shifted in one or more directions from the firstgraphical field of view in response to receiving movement data, from oneor more motion sensing detectors of the IHMD, indicative of the IHMDbeing moved in the one or more directions; and analyzing, by the one ormore processors, the movement data to a determine a user reply to therequest in response to receiving the movement data within apredetermined amount of time.
 9. The computer-implemented method ofclaim 8, further comprising: determining the movement data correspondsto an affirmative head shake; and performing an affirmative actionresponsive to the request.
 10. The computer-implemented method of claim8, further comprising: determining the movement data corresponds to anegative head shake; and performing a negative action responsive to therequest.
 11. The computer-implemented method of claim 8, furthercomprising: determining the movement data does not correspond to anaffirmative head shake or a negative head shake; and generating a secondrequest for the user input.
 12. The computer-implemented method of claim8, wherein the request comprises data requesting an answer to a questionfrom an interaction object.
 13. The computer-implemented method of claim8, wherein the request comprises data requesting to receive or ignorealert data from an electronic appliance.
 14. The computer-implementedmethod of claim 8, wherein the request comprises a modifiable setting ofan electronic appliance.
 15. A non-transitory computer-readable storagemedium including instructions that, upon being executed by one or moreprocessors of an immersive head-mounted display (IHMD), cause the one ormore processors to: generate a first graphical field of view to presenton one or more displays of the IHMD; generate a request for a userinput; generate one or more second graphical fields of view that areshifted in one or more directions from the first graphical field of viewto present on the one or more displays in response to receiving movementdata, from one or more motion sensing detectors of the IHMD, indicativeof the IHMD being moved in the one or more directions; and analyze themovement data to determine a user reply to the request in response toreceiving the movement data within a predetermined amount of time. 16.The non-transitory computer-readable storage medium of claim 15, whereinthe instructions, when executed, further cause the one or moreprocessors to: determine the movement data corresponds to an affirmativehead shake; and perform an affirmative action responsive to the request.17. The non-transitory computer-readable storage medium of claim 15,wherein the instructions, when executed, further cause the one or moreprocessors to: determine the movement data corresponds to an negativehead shake; and perform a negative action responsive to the request. 18.The non-transitory computer-readable storage medium of claim 15, whereinthe instructions, when executed, further cause the one or moreprocessors to: determine the movement data does not correspond to anaffirmative head shake or a negative head shake; and generate a secondrequest for the user input.
 19. The non-transitory computer-readablestorage medium of claim 15, wherein the request comprises datarequesting an answer to a question from an interaction object.
 20. Thenon-transitory computer-readable storage medium of claim 15, wherein therequest comprises data requesting to receive or ignore alert data froman electronic appliance.